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ABSTRACT 

A study examined the levels of comprehension 
generated by questions in story-reiated and story-retell activities 
in the Cooperative Integrated Reading and Composition (CIRC) program. 
A sample of 500 questions from CIRC materials for grades 4, 5, and 6 
were classified. Results indicated that the majority of story-related 
and story-retell questions were classified in the iiteral category, 
although the questions for some stories required a substantial amount 
of inferential comprehension. Results also indicated that even when 
the inferential and evaluation categories were combined, the lower 
level of comprehension represented the largest percentage of 
questions. Findings suggest that writers of CIRC materials should 
make a deliberate attempt to include a balance of literal, 
inferential, and evaluative questions, (Contains 11 references, and 
one table and one figure of data.) (RS) 
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Levels of Reading Comprehension Promoted by the Cooperative Integrated 
Reading and Comprehension (CIRC) Program 

Background and Statement of the Problem 
A major goal of reading instruction, according to the outcomes stated at 
both the state and local levels, is to teach students to perform higher-level 
comprehension tasks such as acquiring and systematically arranging 
information, distinguishing relevant from irrelevant information, deciding how- 
to use data, detecting cause and effect relationships, and the like. In recent 
years, such sophisticated goals added to the notion that reading is a tliinking 
process that cannot be taught effectively through drill and repetitive practice 
alone. Concerning the importance of critical reading skills, Leu and Kinzer 
(1992) wrote: "In reading ... the learner's major objective is to develop, refine, 
and use high-level text thought processes - ultimately, to comprehend" (p. 2). 
Indeed, examination of reading instructional materials reveals that high-level 
comprehension is paramount among learner outcomes. Additionally, 
investigative commissions such as the National Assessment of Educational 
Progress (NAEP, 1987) and Becoming a Nation of Readers (1985) have placed 
heavy emphasis on critical thinking in reading. 

The realization of this goal, however, is doubtful given the fact that 
elementary students receive a meager intellectual diet offering little more than 
decoding and literal comprehension tasks with basal readers. More often than 
not, teachers in elementary schools follow the dictates of published reading 
programs that emphasize isolated skill activities targeting decoding, 
vocabulary, and literal comprehension (Duffy & Mclntyre, 1982; Durkin, 1979; 
Palmer, 1982). As Samuels and Farstrup (1992) staved: 
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Often the teacher's role is primarily that of a technician who follows 
directions and prescriptions, rather than a decision-maker who engages 
in substantive pedagogical maneuvering in response to students' needs. 
However, drill and practice instructional models are inadequate for the 
new comprehension curriculum. This is especially true in a 
technological society - a society that will increasingly value workers who 
can solve problems over those who can follow prescribed routines. It is 
no longer good enough to have students answer literal questions and 
memorize isolated skill responses, (pp. 171-172) 

It is well established in the literature that elementary teachers rely heavily on 
basal readers and the accompanying lesson plans for reading instruction. 
Although implementation differs among teachers, research demonstrates that 
published programs heavily affect classroom practices (Diederich, 1973; 
Durkin, 1984). 

This would not pose a problem with reading instruction if basals 
contained effective lessons which promoted higher-level comprehension. 
However, published reading programs have remained virtually unchanged since 
the 1940s. While the graphics and attractive packaging have improved since 
the early days, basals still emphasize decoding at the expense of 
comprehension. The most traditional lessons employ the Directed-Reading 
Activity format which is skill-driven - that is, the objective of each lesson is to 
learn an isolated skill (which may or may not be relevant to understanding the 
story) rather than to comprehend a particular type of discourse. While 
students are held accountable for learning the skill through independent 
workbook practice and/or end-of-unit tests, they are seldom held accountable 
for the comprehension of the reading selection - except through questioning. 
Most analyses of questioning still indicate that teachers predominantly rely or 
literal or recall-only questions. 
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The weaknesses of basal reader programs prompt teachers to search for 
supplementary or replacement programs, if the school district allows such 
flexibility. The recent development of a cooperative learning program at Johns 
Hopkins University designed especially for reading and writing has been 
received enthusiastically by practitioners who are searching for more effective 
ways to teach reading. The Cooperative Integrated Reading and Composition 
(CIRC) Program (1992) is creating the same excitement among teachers as 
earlier Student Team learning programs because of the positive effects on 
classroom climate, interest in reading, and student achievement (Slavin, 1990). 
A major portion of this program is the story-related activities which include 
Treasure Hunts that teach vocabulary, spelling and story grammar by requiring 
students to respond to questions. These Treasure Hunts may be used in 
conjunction with the story in basal series or with children's literature. After 
reading the story and discussing it in their heterogeneous reading groups, 
students summarize the major events of the story by responding to questions 
known as "Story Retell" with their partners. 

The CIRC Program offers an innovative alternative to basal programs 
that will increase in popularity in coming years. Because of the interest in 
cooperative learning methods in both the professional literature and among 
practitioners, and given the fact that CIRC is a content-specific cooperative 
learning program in reading and writing, the materials should be analyzed to 
determine their emphasis on higher-level comprehension. Specifically, what 
levels of comprehension do most questions in CIRC materials promote? 



Analyzin g CIRC Materia ls 
In an attempt to determine the levels of comprehension generated by 
questions in story-related and story-retell activities in the CIRC program, an 
analysis of questions was completed. The research examined the extent to 
which the CIRC materials emphasized higher-level comprehension. Further, 
this study examined if statistically significant differences existed between the 
number of literal-level questions and above-literal-level questions. It was 
hypothesized that there would be no significant difference between the 
frequencies of story-related and story-retell questions classified at the literal 
level and above-literal-level of comprehension. 

Research Procedures 
Treasure Hunts that accompany each story from children's literature in 
the CIRC materials for grades four, five and six were obtained from the Center 
for Social Organization, Johns Hopkins University. The sample consisted of 
500 questions from the story-related activities and the story-retell section of 
the Treasure Hunts. The 500 questions were randomly selected prior to the 
analysis and extracted from all interrater reliability exercises and discussions. 
The three researchers had received training in how to use CIRC materials in a 
workshop conducted by a consultant from Johns Hopkins University. 
Additionally, both university professors teach Team Learning, Metnods and 
CIRC Methods in methods courses in elementary education and the other 
researcher is a classroom teacher in grade 6 and uses CIRC methods as a 
supplement to her reading program. Because of the researchers' familiarity 
with these materials, the story-related questions and the story-retell questions 
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were of most interest since these questions accompany the reading section of 
children's literature in the CIRC program and would replace the questions that 
accompany a reading selection in basal readers. 

After extensive training in question classification with the levels of 
literal, inferential, and evaluative comprehension and obtaining an interrater 
reliability of .90 on a sample of questions not selected for the actual 
classification study, three raters independently classified 500 questions from 
the story-related or story-retell questions. Results of the classification were 
analyzed and are presented in a. frequency distribution table and bar graph. 
The chi-square statistic was selected to test the proposed hypothesis. 

Findings and Conclusions 
To represent the data, answer the research question, and test the 
proposed hypothesis, the study utilized a bar-graph comparison, frequency 
distribution table, and the chi-square goodness-of-fit analysis. The chi-square 
goodness-of-fit analysis was employed to determine if the observed frequency 
differed form that expected by chance. 

The first research question asked: What levels of questioning are 
promoted by story-related and story-retell questions in CIRC materials? 
In order to answer this research question, the following null hypothesis was 
tested: There is no significant difference between the frequencies of the 
questions classified at the literal-level and the above -literal- le /el. Each 
question randomly selected for classification was rated independently by three 
trained raters. The results of the ratings were compared and the question was 
placed in the category ascribed by the the majority of raters. In the actual 
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classification study, the raters reached total agreement on 90% of the 
questions. Table 1 reports the the frequency distribution of the results of this 
classification. 
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Inspection of these data indicate that the majority of story-related and 
story-retell questions that accompany CIRC materials were classified in the 
literal category. The overall frequency distribution and percentage 
tabulation reveals that of all 500 questions randomly selected for 
analysis, 304 (61%) dealt with literal comprehension. The second largest 
percentage of questions were categorized as inferential 140 (28%). Only 
55 (11 %) of the questions fell into the evaluation category. 
Although these findings indicate that the total observed frequencies were 
largely in the literal category, comparisons among individual Treasure Hunt 
guides yield important differences that should be discerned. 
For example, the data show that some Treasure Hunts contained questions 
that were exclusively literal, while a few Treasure Hunts contained questions 
that required a substantial amount of inferential comprehension. This trend 
of inferential and/ or evaluation questions were represented in the following 
Treasure Hunts: (a) Hello, Mv Name is Scrambled Eggs ; (b) A Thief in the 
Village ; and (c) Anastasia Has the Answers . The Treasure Hunts that 
accompany these novels require comprehension beyond what is stated in the 
text and ask readers to make a critical judgment about information from the 
literature. Closer inspection of these teacher's guides indicates that most 
answers to the questions are listed as "Answers Will Vary" or "Accept any 
Reasonable Response." 

The levels of comprehension in these Treasure Hunts promote critical 
reading and comprehension beyond mere literal events in the story. It is 
unusual; however, that these types of questions would exist in only a few of 
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the many Treasure Hunts analyzed and are so concentrated in these three. It 
is encouraging to know that these types of questions can and do abound in the 
CIRC materials. 

The second research question asked: To what extent do the CIRC 
questions emphasize higher-level comprehension? To answer tnis question, 
the data were dichotomized into two major categories: lower-level and 
higher-level. Lower level comprehension refers to the literal level and 
higher-level comprehension represent the inferential and evaluation categories 
combined. By collapsing the inferential and evaluation categories into one 
major category, it is possible to analyze the emphasis of each question on 
understanding and application of information (higher-level) as compared to 
rote memorization (lower-level). The resulting data were tabulated and the 
percentages were calculated (see Figure 1). These results indicate that, even 
when the inferential and evaluation categories are combined for examination, 
the lower level of comprehension represented the largest percentage of 
questions. It should be noted that the largest percentage of higher 
level questions were judged as inferential (28 %). 

To test the null hypothesis, a chi-square goodness-of-fit analysis was 
calculated to determine if the observed differences between the total 
frequencies of questions (literal level vs. above-literal level) departed 
significantly from those expected by chance. Because the chi-square 
value of 23.33 exceeds the critical chi-square value beyond the .001 level, the 
null hypothesis is rejected. This statistic indicates that the distribution of 
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Figure 1 : Bar Graph Analysis Of The Levels Of Comprehension 
Demonstrated In CIRC Materials 
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total frequencies for the literal-level and above-literal-level questions departed 
significantly from a distribution based on change alone. Therefore, CIRC 
Treasure Hunts include significantly more literal comprehension questions 
than above literal-comprehension questions. 

Implications 

It should be noted that CIRC consists of three main elements: (a) 
Story-Related Activities; (b) Direct Comprehension Skill Lessons; and (c) 
Integration of Language Arts and Writing. Because the story-related activity 
would replace the traditional Directed Reading Lesson in the basal, this 
research investigated the levels of comprehension promoted by the CIRC 
materials (i.e., Treasure Hunts) that accompany the reading of a story from 
children's literature. 

Although the story-related activities are considered the main element in 
these materials, it should be noted that not every aspect of the CIRC program 
was examined in this analysis and that the findings are limited to the 
story-related activities. It is conceivable that higher-level comprehension is 
promoted in the direct comprehension skill lessons or the activities that 
integrate writing. 

These data reveal that students are exposed to a preponderance of literal 
comprehension questions when using the story-related and story-retell 
activities. It is recognized that the acquisition of literal events precedes 
functioning at higher cognitive levels; however, the emphasis given to mere 
recall of story events is excessive for a program of this distinction. If the goal 
. of reading is to promote recall of literal events without interpretation, 



14 



12 

application, or evaluation then the questions contained in the CIRC materials 
will suffice. However, if the goal of reading is higher-level comprehension, then 
the questioning strategies in CIRC story- related materials need to be revised to 
include a balance of literal and higher-level (i.e., inferential and evaluation) 
questions. 

As stated earlier, the type of question used by teachers strongly affects 
what children learn to think about while reading. If literal details receive the 
emphasis in a teacher's questioning, then children learn to attend to details as 
they read (Anderson, 1985). Therefore, repeatedly asking the same type of 
question cues students to focus on the details needed to answer the question, 
and they allocate their attention accordingly. On the other hand, studies (e.g., 
Pearson, 1985) show that making inferences and evaluations results in 
improvement in these types of thinking, without a loss and perhaps even a 
gain, in literal comprehension. Research indicates that the active 
manipulation of literal events enhances retention. 

Data in this study imply that students can answer the questions that 
accompany the story with little or no true understanding of the elements 
involved. Developing higher levels of comprehension while engaging a reader 
with a story requires i more logical progression through all the levels of 
questioning and more attention to inferencing and evaluation. 

The results of this study should be particularly beneficial to writers of 
CIRC materials, especially those involved in developing Treasure Hunts. It is 
recommended that writers make a deliberate attempt within each Treasure 
Hunt to include a balance of literal, inferential, and evaluative questions. 
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